User Guide of RidgeRun GStreamer Analytics
Visualize RidgeRun GStreamer Analytics Metrics with Grafana
Once you have launched the RidgeRun GStreamer Analytics services, you can monitor and visualize system and pipeline performance using Grafana.
Accessing Grafana
Grafana is available locally at:
http://localhost:3000
Log in with your credentials (default user: admin password: admin if unchanged).
Available Dashboards
In Grafana, under Dashboards, you will find the RR Analytics folder. It contains two dashboards:
- GStreamer RR Analytics (Linux Systems): Provides system-level metrics and GStreamer process monitoring for Linux machines.
- GStreamer RR Analytics - Jetson: Provides system-level metrics, GStreamer process monitorings and Jetson-specific metrics, including GPU and accelerators usage.
Dashboard Structure
Metrics are grouped into different sections for easier navigation.
System Metrics
Each registered system has a dedicated row in the dashboard that displays the general metrics, as illustrated below:
As you can see, the panels provide the following measurements:
1. Memory Usage %: percentage of RAM in use.
2. CPU Average %: overall CPU load.
3. Per-core CPU %: utilization per CPU core.
4. Network I/O: sent and received data.
5. Disk I/O: read and write throughput.
6. Memory (Absolute): total amount of memory use
Jetson Metrics (Jetson dashboard only)
On Jetson-based systems, additional hardware-specific metrics are available:
The panels provide the following measurements:
1. Usage %: GPU, EMC, Encoder/Decoder(NVENC, NVDEC, and NVJPG) and Accelerators overall utilization.
2. GPU Frequency: operating frequency of the GPU.
3. EMC Frequency: memory controller (external memory) frequency.
4. Encoder/Decoder Frequency: frequency of NVENC, NVDEC, and NVJPG hardware blocks.
5. Accelerators: frequency of VIC, NVDLA, and PVA accelerators.
Per-Process Metrics
Each GStreamer process can be monitored individually to evaluate resource consumption, stability, and behavior.
The panels provide the following measurements:
1. Process CPU: percentage of CPU resources used by the process.
2. Process I/O: input/output activity attributed to the process.
3. Process Memory %: percentage of system memory consumed by the process.
4. Process Memory (Absolute): total amount of memory used.
5. Runtime: duration of the pipeline execution.
6. Bitrate: data throughput for the pipeline/elements.
7. Framerate: current frames per second processed by the pipeline/elements.
Debug Logging Control
The dashboard includes a controller to manage runtime logging for each system and process. This allows enabling or disabling detailed logs and change the logs levels without restarting services.
The panel provides the following controls:
1. System Selector: choose the system where logs will be controlled.
2. Process Selector: select the specific process within the chosen system.
3. Global Debug Level: set a global logging verbosity level to be stored in the local device file for this process.
4. Filtered Debug Level: apply a filtered debug level that will be streamed into Grafana for analysis. Must be subset of the global debug level.
5. Debug File Size: configure the maximum log file size (in lines, bytes, or time).
6. Refresh Button: update the system/process list, this must be refreshed manually each time a new device is registered.
Interacting with Metrics
Grafana provides tools to refine how metrics are displayed:
1. Adjusting Time Window: Use the top-right time selector (e.g., Last 5 minutes, Last 1 hour) to control the time span of the graphs.
2. Changing Refresh Interval: Next to the refresh button, set how often the data refreshes (e.g., every 5s, 30s, 1m). Use a faster interval for real-time monitoring or longer intervals for historical views.
3. Focusing on One Metric: To isolate one measurement in a panel:
- Click the legend item text (e.g., CPU 0, CPU 1, etc.) to highlight only that series.
- Click again to toggle it back into the view.
Filtering Metrics
The dashboard provides filters at the top of the view, allowing you to narrow down the metrics displayed by System, Process, and Pipeline.
The available filters are:
1. System Selector: choose which registered device (e.g., jetson-orin-agx, jetson-xavier-agx) to display metrics for. You can select more than one system at a time.
2. Process Selector: filter metrics to a specific GStreamer process running on the selected system. You can select more than one process at a time.
3. Pipeline Selector: further refine the view to metrics from a particular pipeline within the process/processes. You can select more than one pipeline at a time.
These filters can be combined to focus only on the data relevant to your analysis (e.g., metrics for a single pipeline on a specific Jetson device).
Logs via Drilldown
Logs from the monitored systems and pipelines are available in the Drilldown → Logs section of Grafana. This view provides real-time access to debug messages captured from each registered device and process.
This view allows you to monitor logs from multiple systems in parallel:
1. Service Panels: each registered device (e.g., jetson-orin-agx, jetson-xavier-agx) shows a summary of log levels (DEBUG, INFO, WARN, etc.) and their counts.
2. Log Streams: detailed logs for each system.
3. Log Counters: quick insight into the number of log entries per severity level for each system.
4. Time Range & Filters: refine which logs are shown by selecting time intervals and label values
When you hit Show logs for specific device, additional details are available:
1. Log Entries: sequential log messages with timestamp, process, level and pipeline/element.
2. Fields Panel: each log can be expanded to show structured metadata extracted from the logs for extra information (file, function, category, etc.).
3. Level/Include/Exclude Filters – refine log visibility by log levels and including or excluding terms.
4. View Modes – logs can be displayed in different formats (text, table, JSON).
Get Per-Process and Per-Pipeline Metrics with Tracers
As you know by now RidgeRun GStreamer Analytics provides a set of tracers that allow monitoring both pipeline-level and process-level performance metrics. These tracers are enabled through the GST_TRACERS environment variable and provide runtime statistics via the GStreamer tracing subsystem.
Available Tracers
| Tracer Name | Description |
|---|---|
rrpipelinebitrate
|
Reports bitrate for the pipeline. |
rrpipelineframerate
|
Reports framerate statistics for the pipeline. |
rrpipelineruntime
|
Reports total runtime of the pipeline. |
rrproccpuusage
|
Reports per-process CPU usage. |
rrprociousage
|
Reports per-process I/O usage (read/write operations). |
rrprocmemusage
|
Reports per-process memory usage. |
General Usage
To enable tracers, set:
GST_TRACERS="<tracer_name>" <GStreamer application>
For gst-launch-1.0:
GST_TRACERS="<tracer_name>" gst-launch-1.0 <pipeline>
You can enable multiple tracers at once by separating them with a semicolon (;):
GST_TRACERS="rrpipelineframerate;rrproccpuusage"
Optionally, you can also set RR_PROC_NAME to assign a human-readable name to the process providing the metrics. If not set, the <process-name>-<PID> is used by default. This makes it easier to identify the source of the metrics in Grafana visualization.
RR_PROC_NAME="encoder_pipeline" GST_TRACERS="rrproccpuusage" gst-launch-1.0 videotestsrc ! videoconvert ! x264enc ! fakesink
Example Pipelines
Process CPU Usage
Measure CPU usage of the process while running a simple encoding pipeline:
GST_TRACERS="rrproccpuusage" gst-launch-1.0 videotestsrc num-buffers=10000 ! 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! videoconvert ! queue ! avenc_h263p ! queue ! avimux ! fakesink
For live sources:
GST_TRACERS="rrproccpuusage" \ gst-launch-1.0 videotestsrc is-live=true num-buffers=10000 ! \ 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! \ videoconvert ! queue ! avenc_h263p ! queue ! avimux ! fakesink
You can observe that the CPU usage of the first example is typically higher than the live source pipeline, since data is pushed as fast as possible rather than in real-time.
Process Memory Usage
Monitor memory usage of the process:
GST_TRACERS="rrprocmemusage" \ gst-launch-1.0 videotestsrc num-buffers=10000 ! \ 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! \ videoconvert ! queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 ! \ avenc_h263p ! queue ! avimux ! fakesink
You may notice that memory usage can increase depending on the queue configuration. For example, disabling queue limits (as shown in the example) may cause higher memory consumption since buffers accumulate without restriction.
Process I/O Usage
Track read/write usage at the process level.
For read operations:
GST_TRACERS="rrprociousage" \ gst-launch-1.0 filesrc location=test.avi ! \ decodebin ! perf ! fakesink sync=true
For write operations:
GST_TRACERS="rrprociousage" \ gst-launch-1.0 videotestsrc is-live=true num-buffers=10000 ! \ 'video/x-raw,format=YUY2,width=1280,height=720,framerate=30/1' ! \ videoconvert ! x264enc ! queue ! avimux ! filesink location=test.avi
When using a file source, I/O usage reflects the read operations from disk, while in the synthetic source case the I/O activity is mainly due to writing the encoded stream to the output file.
Pipeline Metrics
You can enable all pipeline tracers together to monitor bitrate, framerate, and runtime in a single execution:
RR_PROC_NAME=pipeline-metrics GST_TRACERS="rrpipelinebitrate;rrpipelineframerate;rrpipelineruntime" \ gst-launch-1.0 videotestsrc is-live=true ! \ 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! \ videoconvert ! videorate drop-only=true max-rate=15 ! queue ! x264enc ! fakesink
This pipeline will report bitrate, framerate, and runtime statistics simultaneously, allowing you to correlate how encoding settings and videorate affect each metric.
Capture and Visualize GStreamer Debug Logs with RidgeRun Log Tracer
Unlike the metric-oriented tracers, rrlogtracer is designed to capture and forward detailed debug logs from a GStreamer pipeline.
Log Tracer Usage
To enable the log tracer, set:
GST_TRACERS="rrlogtracer" <GStreamer application>
For example, with gst-launch-1.0:
GST_TRACERS="rrlogtracer" gst-launch-1.0 <pipeline>
You can also configure additional environment variables to control how logs are captured and stored.
The following example shows rrlogtracer with a live source and custom settings:
RR_PROC_NAME="my-process" \ RR_RING_MODE=bytes \ RR_RING_LIMIT=5M \ RR_RING_REMOTE_MODE=lines \ RR_RING_REMOTE_LIMIT=10000 \ GST_DEBUG="DEBUG" \ GST_REMOTE_DEBUG="DEBUG" \ GST_TRACERS="rrlogtracer" \ gst-launch-1.0 videotestsrc is-live=true ! \ identity silent=true name=tap ! queue ! videoconvert ! fakesink sync=false
See Log Tracer Environment Variables for details on each configurable setting.
Log Tracer Environment Variables
The behavior of rrlogtracer can be controlled using the following environment variables:
| Variable | Description | Accepted Values | Default |
|---|---|---|---|
RR_PROC_NAME
|
Name assigned to the process. Used to identify log files. If not set, PID-based process name is used (e.g., gst-launch-1.0-600).
|
Any string | Process name - PID |
RR_RING_MODE
|
Mode for the in-memory ring buffer. | bytes | lines | time
|
bytes
|
RR_RING_LIMIT
|
Maximum size of the ring buffer. | For bytes: e.g. 128K, 10M, 1G, 1048576For lines: number of linesFor time: duration (e.g. 5s, 2m)
|
10MiB
|
RR_RING_REMOTE_MODE
|
How logs are sent to remote consumers. | bytes | lines | time
|
bytes
|
RR_RING_REMOTE_LIMIT
|
Maximum size/entries for the remote buffer. | Same format as RR_RING_LIMIT depending on mode
|
10MiB
|
GST_DEBUG
|
Sets the verbosity level for debug messages. | Standard GStreamer debug levels (e.g. ERROR, WARNING, INFO, DEBUG, LOG, TRACE)
|
|
GST_REMOTE_DEBUG
|
Sets the verbosity level for remote/filtered debug messages. | Same as GST_DEBUG
|
Log Tracer Files
When running your GStreamer application with rrlogtracer, two log files are generated under /var/log/ridgerun:
- gst-<process-name>.log
- gst-<process-name>-filtered.log
where <process-name> corresponds to the name defined in RR_PROC_NAME environment variable.
If RR_PROC_NAME is not set, the process name defaults to the GStreamer application name plus its PID (e.g., gst-launch-1.0-600).